Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

The issue fixed

There hasn't been an issue on this.

In the past there was no possiblity of making attention_mask and past_kv_cache work together. If attention_mask is made the same length as (cached tokens + new tokens) it cannot bypass the shape assertion. If attention_mask is made the same length as new tokens the model cannot attend to cached tokens in attention layers.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Change summary:

In the past the HookedTransformer cannot infer with past_kv_cache and attention_mask both enabled. This is because in the preprocessing stage some branching criterion is not correct.

I fix that by rearranging the branches. I also added a unit test for this, in tests/integration/test_kv_cache.py::test_kv_cache_with_custom_attention_mask.

In detail:

The red part should also apply to where the green arrow points to. I did not simply copy the code twice. I rearranged the branches. My implementation introduces some extra assert in certain cases but it has not failed any existing tests.

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility